Display system

ABSTRACT

A display system changes display in accordance with a gesture of an operator in a vehicle. The display system includes a gesture detector that detects the gesture, a vision area detector that detects a vision area of the operator, a display that displays a gesture operation result that is an operation result of the gesture, and a display controller that executes a control related to display processing of the display. The display controller acquires the vision area detected by the vision area detector and a display area of the display, determines a positional relationship between the vision area and the display area, and displays the gesture operation result to the operator. The display controller determines a display mode of the gesture operation result based on the positional relationship between the vision area and the display area.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of InternationalPatent Application No. PCT/JP2022/002837 filed on Jan. 26, 2022, whichdesignated the U.S. and claims the benefit of priority from JapanesePatent Application No. 2021-017659 filed on Feb. 5, 2021. The entiredisclosures of all of the above applications are incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to a display system.

BACKGROUND

A vehicle information display device displays feedback information thatis information in response to recognition of a gesture input.

SUMMARY

According to at least one embodiment of the disclosure, a display systemchanges display in accordance with a gesture of an operator in avehicle. The display system includes a gesture detector configured todetect the gesture, a vision area detector configured to detect a visionarea of the operator, a display configured to display a gestureoperation result that is an operation result of the gesture, and adisplay controller configured to execute a control related to displayprocessing of the display. The display controller includes an areaacquisition unit configured to acquire the vision area detected by thevision area detector and a display area of the display, a positiondetermination unit configured to determine a positional relationshipbetween the vision area and the display area, and a display executionunit configured to display the gesture operation result to the operator.The display execution unit determines a display mode of the gestureoperation result based on the positional relationship between the visionarea and the display area.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an arrangement of a display.

FIG. 2 is a block diagram related to control of a display system.

FIG. 3 is a diagram illustrating a vision area of a driver lookingahead.

FIG. 4 is a flowchart illustrating the control of the display system.

FIG. 5 is a diagram illustrating a vision area of the driver looking ata center information display.

FIG. 6 is a diagram illustrating a vision area of the driver looking ata meter display.

FIG. 7 is a block diagram related to control of a display systemaccording to a second embodiment.

FIG. 8 is a flowchart illustrating the control of the display systemaccording to the second embodiment.

FIG. 9 is a diagram illustrating a vision area of the driver looking atan electronic mirror.

DETAILED DESCRIPTION

A comparative example will be described first. A vehicle informationdisplay device according to the comparative example displays feedbackinformation that is information in response to recognition of a gestureinput. The disclosure of JP 2018-103646 A is incorporated herein byreference to explain technical elements presented herein.

In the configuration of the comparative example, the feedbackinformation is displayed at different positions on a head-up displayaccording to a driving situation. Accordingly, a driver can concentrateon driving during vehicle traveling, while visibility of the feedbackinformation is enhanced during vehicle stopping. However, an operatorperforming a gesture does not always look an area including the head-updisplay. If the driver does not look the head-up display, the driver whohas performed the gesture cannot confirm the feedback information.Therefore, when an operation result is different from an intention ofthe operator who has performed the gesture, the operator may bedifficult to notice that the operation is different from his or herintention.

In contrast to the comparative example, according to the presentdisclosure, a display system can display an operation result of agesture to be easily recognizable.

According to an aspect of the disclosure, a display system changesdisplay in accordance with a gesture of an operator in a vehicle. Thedisplay system includes a gesture detector configured to detect thegesture, a vision area detector configured to detect a vision area ofthe operator, a display configured to display a gesture operation resultthat is an operation result of the gesture, and a display controllerconfigured to execute a control related to display processing of thedisplay. The display controller includes an area acquisition unitconfigured to acquire the vision area detected by the vision areadetector and a display area of the display, a position determinationunit configured to determine a positional relationship between thevision area and the display area, and a display execution unitconfigured to display the gesture operation result to the operator. Thedisplay execution unit determines a display mode of the gestureoperation result based on the positional relationship between the visionarea and the display area.

According to the display system, the display execution unit determinesthe display mode of the gesture operation result based on the positionalrelationship between the vision area and the display area. Therefore,the display mode of the gesture operation result can be changed inaccordance with the position of the vision area that may changedepending on the state of the operator. Therefore, according to thedisplay system, an operation result of the gesture can be easilychecked.

According to an aspect of the disclosure, a method is used for a displaysystem configured to change display of a display in accordance with agesture of an operator detected by a gesture detector. The methodincludes acquiring a vision area of the operator detected by a visionarea detector and a display area of the display capable of displaying agesture operation result which is an operation result of the gesture,determining a positional relationship between the vision area and thedisplay area, and displaying the gesture operation result to theoperator. The displaying includes determining a display mode of thegesture operation result based on the positional relationship betweenthe vision area and the display area.

According to the method, the display mode of the gesture operationresult is determined based on the positional relationship between thevision area and the display area. Therefore, the display mode of thegesture operation result can be changed in accordance with the positionof the vision area that may change depending on the state of theoperator. Therefore, according to the method, an operation result of thegesture can be easily checked.

Hereinafter, embodiments will be described with reference to thedrawings. In the embodiments, parts that functionally and/orstructurally correspond to or are associated with each other may beassigned the same reference numeral, or reference numerals different indigit in the hundreds or higher place. With respect to the parts thatcorrespond to or are associated with each other, explanations thereofcan be shared among the embodiments.

First Embodiment

In FIG. 1 , a display 10 constituting a display system 1 is mounted on avehicle. The display system 1 provides various types of information toan occupant by displaying information related to traveling of a vehicle,information for providing entertainment, for example. However, thedisplay system 1 is configured to use not only visual information butalso non-visual information such as audio information. The displaysystem 1 is configured to use visual information and non-visualinformation in combination. A display mode includes various items suchas a content to be displayed on the display system 1, a display positionof the content, and a display timing of the content. Therefore,determining the display mode can mean determining each of the items suchas the content to be displayed.

The display 10 includes a head-up display 11, a display unit 13, andelectronic mirrors 15. However, the display 10 does not need to includeall the devices described above, and may include other devices notdescribed above.

The head-up display 11 displays a virtual image superimposed on a frontwindow. The head-up display 11 includes a body portion 11 h and adisplay portion 11 d. The head-up display 11 executes necessary displayby projecting an image from the body portion 11 h toward the displayportion 11 d.

The head-up display 11 mainly executes display for making the driveraware of information related to vehicle traveling. For example, a routeguidance display, a warning display, or the like is displayed on thehead-up display 11 as necessary. The display of the head-up display 11can be recognized by the driver even in a situation in which the driveris looking ahead to drive the vehicle. The display portion 11 d of thehead-up display 11 is provided at a position in front of the driver'sseat. The head-up display 11 provides an example of a small display.

The display unit 13 displays necessary information mainly to an occupantseated in a front seat of the vehicle. The display unit 13 includes, forexample, a liquid crystal display or an organic electroluminescentdisplay. The display unit 13 has a wider displayable area and higherresolution than the head-up display 11. Therefore, the display unit 13can widely display a finer image than the head-up display 11. Therefore,the display unit 13 is capable of displaying more information than thehead-up display 11. The display unit 13 provides an example of a largedisplay.

The display unit 13 is provided on an instrument panel of the vehicle.Therefore, the display unit 13 can be provided at a position lower thanthe display portion 11 d of the head-up display 11. The display unit 13extends along a right-left direction of the instrument panel, and isprovided to be horizontally long so as to include a position in front ofthe driver's seat and a position in front of the passenger's seat.

The display unit 13 includes display screens and can simultaneouslydisplay multiple pieces of information of different types. The displayunit 13 displays information related to a state of the vehicle, such asvehicle speed, a remaining fuel amount, and a remaining battery amount.The display unit 13 includes a meter display 13 m on which informationabout the state of the vehicle is displayed. The meter display 13 m isprovided in a region of the instrument panel located frontward of asteering wheel.

The display unit 13 displays, for example, information such as routeguidance related to a car navigation device and reproduced moving imagesof video contents. The display unit 13 includes infotainment display 13i on which information such as route guidance is displayed. Theinfotainment display 13 i can also be a device that provides informationand entertainment to an occupant. The infotainment display 13 i isprovided mainly at a position in front of the passenger seat in theinstrument panel.

The display unit 13 displays, for example, information on a state of anair conditioner, a state of an audio device, or the like. The displayunit 13 includes a center information display 13 c on which informationon the state of the air conditioner or the like is displayed. The centerinformation display 13 c is provided in a center cluster of theinstrument panel.

The center information display 13 c displays information that can bechanged by an operation of the occupant. The information that can bechanged by the operation of the occupant is, for example, informationsuch as a set temperature of air conditioning, a set air volume of theair conditioning, and a volume of the audio device.

The center information display 13 c includes a touch panel. Therefore,the center information display 13 c has two functions. One is a functionof displaying information, and the other is a function of receiving anoperation from the occupant. In other words, the center informationdisplay 13 c functions not only as the display 10 but also as a gesturedetector 53 described later. The touch panel of the center informationdisplay 13 c detects a gesture as a motion of the occupant performed ina state of being in contact with the panel. Examples of specificgestures will be described later.

The touch panel of the center information display 13 c can also acceptoperations other than gestures. For example, an operation can also beaccepted by pressing an operation button on the touch panel. The centerinformation display 13 c may also be an operation display for receivingan operation by the occupant. On the other hand, the meter display 13 mand the infotainment display 13 i may be display-only displays thatexecutes display only.

The gesture detectable by the center information display 13 c is notlimited to an operation involving contact with the touch panel. Forexample, the center information display 13 c may include an opticalsensor that detects reflected light or transmitted light of projectedlight, and may detect a gesture from a change in light due to a motionof the occupant. For example, the center information display 13 c mayinclude a capacitance sensor that detects capacitance, and may detect agesture from a change in capacitance due to a motion of the occupant.Therefore, the center information display 13 c may adopt a configurationin which a contactless operation not involving contact with the centerinformation display 13 c is detected as a gesture.

The meter display 13 m and the infotainment display 13 i are next toeach other in the right-left direction. A display screen for displayinginformation such as current time and weather may be provided between themeter display 13 m and the infotainment display 13 i. The infotainmentdisplay 13 i and the center information display 13 c are next to eachother in the up-down direction. The center information display 13 c isprovided below the infotainment display 13 i. Therefore, when operatingthe center information display 13 c with a finger, the occupant does notblock the display screen of the infotainment display 13 i with his orher hand or finger.

The display unit 13 is a pillar-to-pillar type display extending from anedge adjacent to a base of a left A-pillar to an edge adjacent to a baseof a right A-pillar. The display unit 13 is formed by one display inwhich multiple display screens showing different display contents arecontinuous without a seam. In other words, a part of the one continuousdisplay functions as the meter display 13 m, another part functions asthe infotainment display 13 i, and another part functions as the centerinformation display 13 c. In the display unit 13, it is possible toarbitrarily set what content is displayed and where the content isdisplayed. For example, a portion functioning as the center informationdisplay 13 c may be positioned in a front area of the front passengerseat instead of the center cluster of the instrument panel.

The display unit 13 is not limited to a case of being formed by the onecontinuous display. For example, the display unit 13 may be formed as awhole by multiple displays arranged on the instrument panel so as to beseparated from each other.

The electronic mirrors 15 function as a rearview mirror and shows animage captured by a camera device on a display instead of reflectinglight on a mirror attached to a vehicle. The electronic mirrors 15functioning as left and right mirrors take images in respective views ofleft and right rear sides of the vehicle with the camera device anddisplays the images on the displays. The electronic mirrors 15 may be aperiphery monitoring device for monitoring the periphery of the vehicle.

The electronic mirrors 15 are formed by displays separate from thedisplay unit 13. However, the electronic mirrors 15 may be formed as adisplay integrated with the display unit 13.

The display portion 11 d of the head-up display 11 provides an exampleof a display area. The display screen of the display unit 13 provides anexample of the display area. The display screens of the electronicmirrors 15 provide an example of the display area. In summary, thedisplay area includes the display portion 11 d and the multiple displayscreens.

In FIG. 2 , the display system 1 includes the display 10 and a displaycontroller 70. The display controller 70 is connected to a vision areadetector 51. The vision area detector 51 detects a vision area VA whichis an area visually recognized by an operator such as a driver of thevehicle. Hereinafter, a case where the operator is the driver of thevehicle will be described as an example. However, an occupant other thanthe driver may be set as the operator. The vision area detector 51 is,for example, an in-vehicle camera that captures an image of the insideof the vehicle including the eyes of the driver. The vision areadetector 51 functions as a device for grasping a state of the driver,that is, functions as a line-of-sight detection sensor of a driverstatus monitor.

The vision area VA of the driver looking ahead will be described withreference to FIG. 3 . In the drawing, the vision area VA is indicated bydot hatching. The center position of the vision area VA is located abovethe display portion 11 d of the head-up display 11. The center positionof the vision area VA can also be referred to as a line-of-sightposition. The vision area VA includes the display portion 11 d of thehead-up display 11. Therefore, the driver can check the contentdisplayed on the display portion 11 d of the head-up display 11 whilelooking ahead. The vision area VA does not include a display area otherthan the display portion 11 d of the head-up display 11.

The vision area VA is formed in a substantially circular shape with thevisual line-of-sight position as the center position. In addition, thevision area VA tends to become narrower as the vehicle speed increases.Therefore, the vision area detector 51 detects the position and the sizeof the vision area VA using information of the line-of-sight positionand information of the vehicle speed.

In FIG. 2 , the display controller 70 is connected to the display 10including the head-up display 11, the display unit 13, and theelectronic mirrors 15. The display controller 70 controls the display 10to execute necessary display processing for the occupant. In thedrawings, the meter display 13 m is denoted as MET, the infotainmentdisplay 13 i is denoted as IVI, and the center information display 13 cis denoted as CID. Here, IVI is an abbreviation for In-VehicleInfotainment. The display controller 70 may include a processor and amemory storing a program to be executed by the processor to achieve thedisplay processing of the display 10.

The display controller 70 is connected to the gesture detector 53 of thecenter information display 13 c. The gesture detector 53 detects agesture as a touch operation of the occupant on the touch panel or anoperation of the occupant on the non-contact sensor. The displaycontroller 70 controls the gesture detector 53 to detect a gestureperformed by the occupant.

The display controller 70 is connected to a speaker 21. The speaker 21is a device capable of outputting sound into the vehicle. The speaker 21is capable of outputting, to the inside of the vehicle, a sound thatgives notification to the occupant, such as voice guidance in routeguidance. In addition, the speaker 21 is capable of outputting, to theinside of the vehicle, a sound that provides amusement to the occupant,such as music or sound of video. The display controller 70 controls thespeaker 21 to output sound to the inside of the vehicle.

The display controller 70 includes a gesture acquisition unit 71, anarea acquisition unit 72, a position determination unit 73, and adisplay execution unit 74. The gesture acquisition unit 71 acquires agesture detected by the gesture detector 53 as a gesture operationresult. The gesture operation result is a result of a change made to anoperation target device such as an air conditioner or an audio device bya gesture of the occupant. The gesture operation result is, for example,to lower the set temperature of cooling by 2° C. The gesture operationresult is, for example, to set the volume of music to zero.

The area acquisition unit 72 acquires the vision area VA of the occupantdetected by the vision area detector 51. The area acquisition unit 72acquires the display area by recognizing the positions of the displayportion 11 d of the head-up display 11, the display screen of thedisplay unit 13, and the display screens of the electronic mirrors 15.Information on the display area includes information on the positions ofthe display screen of the meter display 13 m, the display screen of theinfotainment display 13 i, and the display screen of the centerinformation display 13 c on the display screen of the display unit 13.

The position determination unit 73 determines a positional relationshipbetween the vision area VA and the display area acquired by the areaacquisition unit 72. More specifically, it is determined whether thedisplay area is included in the vision area VA. When even a part of thedisplay area is included in the vision area VA, it is determined whichdisplay screen of the display 10 corresponds to the display areaincluded in the vision area VA.

The display execution unit 74 executes a display control in which thegesture operation result is fed back as a gesture response to anoperator who has performed a gesture. The display control includes acontrol in which display on the display 10 is executed and a control inwhich sound is output from the speaker 21.

A flow of the display control for a gesture will be described withreference to FIG. 4 . The display control for a gesture is started whenpower is supplied to the vehicle, for example, when a power switch ofthe vehicle is turned on by an occupant. When the display control for agesture is started, in step S101, the gesture detector 53 startsreceiving a gesture performed by the occupant. After the receiving ofthe gesture is started, the control proceeds to step S111.

In step S111, the display controller 70 determines whether there is aninput of a gesture by the occupant. Examples of the gesture include anoperation of tracing the touch panel of the center information display13 c in the right-left direction, and an operation of tracing the touchpanel in the up-down direction. In this case, different gestures can beset for respective items to be changed. For example, the operation oftracing the touch panel in the right-left direction may correspond tochange in set air volume, and the operation of tracing the touch panelin the up-down direction may correspond to change in set temperature.

In addition, the number of times of the gesture or the range of thegesture may be linked to a magnitude of a degree of the change in settemperature or the like. For example, an operation of tracing the touchpanel downward twice may be set as a gesture of lowering the settemperature of cooling by 2° C. For example, an operation of tracing thetouch panel downward by 2 cm may be set as a gesture of lowering the settemperature of cooling by 2° C. When it is determined in step S111 thatthere is a gesture input, the control proceeds to step S112. On theother hand, when there is no gesture input at step S111, it isdetermined that there is no information to be displayed regarding agesture, and the control proceeds to step S191.

In step S112, the display controller 70 determines whether the vehicleis stopped. Whether the vehicle is stopped can be determined from a gearshift position of the vehicle, a vehicle speed detected by a vehiclespeed sensor, or the like. When the vehicle is determined not to bestopped, i.e., determined to be traveling at step S112, the controlproceeds to step S121. On the other hand, when the vehicle is determinedto be stopped at S112, it is determined that a necessity for the driverto look ahead is low, and the control proceeds to step S161.

In step S121, the area acquisition unit 72 acquires the vision area VAof the driver. FIG. 5 shows the vision area VA when the driver is gazingat the center information display 13 c. The center position of thevision area VA is located on the display screen of the centerinformation display 13 c. The vision area VA includes not only thecenter information display 13 c but also a part of the infotainmentdisplay 13 i. The control in step S121 provides an example of an areaacquisition step.

FIG. 6 shows the vision area VA when the driver is gazing at the meterdisplay 13 m. The center position of the vision area VA is located onthe display screen of the meter display 13 m. The display area includedin the vision area VA is only the display screen of the meter display 13m. The area acquisition unit 72 acquires the vision area VA and acquiresthe latest display area. After the vision area VA and the latest displayarea are acquired, the control proceeds to step S122.

In step S122, the position determination unit 73 determines whether thedisplay area is included in the vision area VA. A situation in which thedisplay area is not included in the vision area VA is assumed to be, forexample, a situation in which the driver is gazing in the right-leftdirection for checking right and left sides of the vehicle.Alternatively, the situation in which the display area is not includedin the vision area VA is assumed to be, for example, a situation inwhich the driver looks ahead without the display portion 11 d of thehead-up display 11 being not included in the vision area VA, forexample, when looking up at a traffic light ahead. The control in stepS122 provides an example of a position determination step.

When even a part of the display area is determined to be included in thevision area VA in step S122, the control proceeds to step S131. On theother hand, when the display area is determined not to be included inthe vision area VA in step S122, it is determined that the driver cannotimmediately visually recognize information displayed by the display 10,and the control proceeds to step S151.

In step S131, the position determination unit 73 determines whether thedisplay screen of the center information display 13 c is included in thevision area VA. When the center information display 13 c is determinedto be included in the vision area VA in step S131, the control proceedsto step S161. On the other hand, when the display screen of the centerinformation display 13 c is determined not to be included in the visionarea VA in step S131, the control proceeds to step S141. The control ofstep S131 provides an example of a position determination step.

In step S141, the display execution unit 74 displays the gestureoperation result on the display screen included in the vision area VA.For example, when the vision area VA is located on the display portion11 d of the head-up display 11, the gesture operation result isdisplayed on the display portion 11 d. Here, the head-up display 11 is adevice in which contents to be displayed for respective display sectionsin the display portion 11 d are set in advance. For example, the routeguidance display and the gesture operation result are displayed indifferent display sections in the display portion 11 d. Therefore, thegesture operation result may be displayed in a display section dedicatedfor displaying the gesture operation result in the display portion 11 d.The control of step S141 provides an example of a display executionstep.

When the center position of the vision area VA is located on a centerscreen of the infotainment display 13 i, the gesture operation result isdisplayed on the center screen of the infotainment display 13 i insteadof a left screen or a right screen of the infotainment display 13 i.Here, the infotainment display 13 i is a device capable of displayingarbitrary content at an arbitrary place on the display screen.Therefore, the gesture operation result is displayed in an areaincluding the center position of the vision area VA in the center screenof the infotainment display 13 i.

In the infotainment display 13 i, a part of the display screen may beset in advance as a display section capable of displaying the gestureoperation result. For example, a right display section may be providedin a part of the right screen, a left display section may be provided ina part of the left screen, and a center display section may be providedin a part of the center screen. In this case, the gesture operationresult is displayed in a display section closest to the center positionof the vision area VA among the right display section, the left displaysection, and the center display section provided apart from each other.

When the vision area VA is located on the display screen of the meterdisplay 13 m, the gesture operation result is displayed on the displayscreen of the meter display 13 m. When the vision area VA is located onthe display screen of one of the electronic mirrors 15, the gestureoperation result is displayed on the display screen of the one of theelectronic mirrors 15. At this time, the gesture operation result isdisplayed on the one of the electronic mirrors 15 included in the visionarea VA, and the gesture operation result is not displayed on the otherof the electronic mirrors 15 that is not included in the vision area VA.

When the gesture operation result is displayed on the display portion 11d of the head-up display 11, the gesture operation result is displayedas an icon to the driver. More specifically, the gesture operationresult is displayed as an icon indicating that the setting of theoperation target device has been changed by the gesture. The icon issimple and expresses information in an intuitively understandablemanner. Accordingly, the driver can visually recognize the icondisplayed on the display portion 11 d of the head-up display 11 andrecognize that the setting of the operation target device has beenchanged by the gesture.

When the gesture operation result is displayed on the infotainmentdisplay 13 i, the gesture operation result is displayed in detail as adetailed information to the driver. The displayed detailed informationhas a larger amount of information than the icon. The displayed detailedinformation includes information on what content the setting of theoperation target device has been changed to. For example, it isdisplayed that the set temperature of the cooling is lowered by 2° C.Alternatively, it is displayed that the set temperature after the changehas been changed from 28° C. to 26° C. After the gesture operationresult is displayed on the display screen in the vision area VA in stepS141, the control proceeds to step S191.

In step S151, the display execution unit 74 outputs the gestureoperation result by sound. For example, a fact that the set temperatureof the cooling is lowered by 2° C. is output by voice from the speaker21. Alternatively, a sound indicating that the set temperature has beenchanged from 28° C. to 26° C. is output from the speaker 21.Alternatively, only a change of the set temperature of the cooling isoutput from the speaker 21 as a notification sound. After outputting thegesture operation result by sound at S151, the control proceeds to stepS191.

In step S161, the display execution unit 74 displays the gestureoperation result on the center information display 13 c. The centerinformation display 13 c displays detailed information instead of anicon. Here, the displayed detailed information may be different from thedetailed information displayed on the infotainment display 13 i. Forexample, as the detailed information displayed on the center informationdisplay 13 c, only the final result changed by the gesture may bedisplayed. More specifically, it may be displayed that the settemperature of the cooling is 26° C. After the detailed information isdisplayed on the center information display 13 c in step S161, thecontrol proceeds to step S191. The control of the step S161 provides anexample of a display execution step.

In step S191, the display controller 70 determines whether the receivingof the gesture has ended. For example, when the power switch of thevehicle is turned off, the gesture receiving has ended. When thereceiving of the gesture has ended, it is determined that there is nopossibility that a new gesture is input by the occupant, and the displaycontrol for the gesture ends. On the other hand, when the receiving ofthe gesture is maintained, it is determined that there is a possibilitythat a new gesture is input by the occupant, and the control returns tostep S111 to repeat the series of display controls for the gesture.

The following will describe technical effects of the present embodiment.According to the embodiment described above, the display execution unit74 determines a display mode of the gesture operation result based onthe positional relationship between the vision area VA and the displayarea. Therefore, it is easy to appropriately respond to a gesture of theoperator as compared with a case where display is always executed in thesame manner regardless of the position of the vision area VA that maychange depending on the situation of the operator. Therefore, it is easyfor the operator to grasp whether the operation by the gesture isappropriately reflected in the control of the operation target device.Therefore, in the display system 1, the operation result of the gesturecan be easily checked by the operator.

In particular, the gesture is an operation method that can be executedwithout looking at an operation screen or the like, and is considered tobe an operation that is highly likely to be executed while the operatorlooking ahead. Therefore, if the gesture operation result is alwaysdisplayed on the center information display 13 c, it is necessary tocheck the gesture operation result at a timing at which the operator isunnecessary to look ahead, in other words, a timing at which the gestureoperation result can be checked is limited. In addition, when thegesture operation result is always output as a sound, it may bedifficult to hear the gesture operation result due to another sound, forexample, during route guidance or near a railroad crossing. Therefore,an optimal display mode is automatically selected based on thepositional relationship between the vision area VA and the display area,and then the gesture operation result is displayed. This automaticselection is useful when transmitting the gesture operation result tothe driver of the vehicle who needs to look ahead.

When at least a part of the display area is included in the vision areaVA (YES in step S122), the display execution unit 74 displays thegesture operation result in the display area within the vision area VA.On the other hand, when the display area is not included in the visionarea VA (NO in step S122), the gesture operation result is not displayedin the display area. Therefore, it is easy to reduce a frequency ofoccurrence of a situation in which the displayed gesture operationresult is not visually recognized by the operator. In addition, it ispossible to reduce a frequency of display of the gesture operationresult in a situation where the operator cannot immediately visuallyrecognize the display.

The display execution unit 74 displays the gesture operation result at aposition including the center position of the vision area VA. Therefore,it is possible to display the gesture operation result at a positionthat is particularly easy to be visually recognized in the vision areaVA.

The display execution unit 74 displays the gesture operation result in adisplay section located at a position closest to the center position ofthe vision area VA among the display sections. Therefore, it is possibleto display the gesture operation result at a position that isparticularly easy to be visually recognized among the display sections.

The gesture operation result displayed on the head-up display 11 isdifferent from the gesture operation result displayed on the displayunit 13. In other words, the display execution unit 74 displaysdifferent contents between the case of displaying the gesture operationresult on the small display and the case of displaying the gestureoperation result on the large display. Therefore, it is possible toselect an appropriate display content according to the size of thedisplay area of the display 10 and display the gesture operation resultvia the selected content.

The display execution unit 74 displays an icon when the gestureoperation result is displayed using the head-up display 11, and displaysdetailed information when the gesture operation result is displayedusing the display unit 13. In other words, the display execution unit 74displays an icon when the gesture operation result is displayed usingthe small display, and displays detailed information when the gestureoperation result is displayed using the large display. Therefore, sincethe head-up display 11 is smaller in display area and is likely to belower in resolution than the display unit 13, it is possible to simplifythe content displayed on the head-up display 11. Therefore, it is easyto reduce a frequency of occurrence of a situation in which the displayof the gesture operation result displayed on the head-up display 11 istoo complicated to be visually recognized by the operator.

When the center information display 13 c and the infotainment display 13i are included in the vision area VA, the display execution unit 74displays the gesture operation result on the center information display13 c. In other words, when the center information display 13 c isincluded in the vision area VA (YES in step S131), the display executionunit 74 displays the gesture operation result on the center informationdisplay 13 c in priority to the other displays. Here, the centerinformation display 13 c includes the touch panel that functions as thegesture detector 53. Therefore, it is easy to correct the operation ofthe operation target device by additionally performing a gesture afterconfirming the gesture detection result.

When the vehicle is stopped (YES in S112), the display execution unit 74displays the gesture operation result on the center information display13 c regardless of the positional relationship between the vision areaVA and the display area. Therefore, in a situation where the centerinformation display 13 c can be easily visually recognized, the gestureoperation result can be displayed on the center information display 13 cregardless of the line of sight of the operator. Therefore, the timerequired from the performance of the gesture to the display of thegesture operation result can be easily shortened. The center informationdisplay 13 c includes the touch panel. Therefore, it is easy to correctthe operation of the operation target device by additionally performingthe touch operation after confirming the gesture detection result.

When the display area is not included in the vision area VA (NO in stepS122), the display execution unit 74 does not display the gestureoperation result using the display 10 and outputs the gesture operationresult from the speaker 21 as a sound. Therefore, unnecessary display ina display area that is not visually recognized by the operator can bereduced, and the gesture operation result can be notified by sound.Therefore, as compared to a case where the notification of the gestureoperation result is suspended until the display area is included in thevision area VA, it is easy to quickly notify the operator of the gestureoperation result. Therefore, it is easy to prevent the operator frommisunderstanding that the operation by the gesture is not received.

In the display execution step, the display mode of the gesture operationresult is determined based on the positional relationship between thevision area VA and the display area. Therefore, it is easy toappropriately respond to a gesture of the operator as compared with acase where display is always executed in the same manner regardless ofthe position of the vision area VA that may change depending on thesituation of the operator. Therefore, it is easy for the operator tograsp whether the operation by the gesture is appropriately reflected inthe control of the operation target device. Therefore, it is possible toprovide a display control method that is easy to operate using agesture.

Second Embodiment

This embodiment is a modification of the basic preceding embodiment. Inthis embodiment, when one of display screens of electronic mirrors 15 isincluded in a vision area VA, display of the gesture operation result isstopped.

In FIG. 7 , a display controller 70 is connected to a gesture detector253. The gesture detector 253 includes an in-vehicle camera thatcaptures an image of an occupant, and detects a gesture from thecaptured motion of the occupant. The display controller 70 acquires thegesture detected using the gesture detector 253.

The display controller 70 is connected to a communication device 255.The communication device 255 enables communication between a mobileterminal 256 carried by the occupant and the display controller 70. Thedisplay controller 70 performs necessary display related to the mobileterminal 256 via the communication device 255. For example, the displaycontroller 70 displays on the display 10 that there is an incoming callto the mobile terminal 256, and provides the occupant with a proposal tostart a hands-free call.

A display control for a gesture will be described with reference to FIG.8 , in which a hands-free call using the mobile terminal 256 is taken asan example. The display control for a gesture is started, for example,when there is an incoming call to the mobile terminal 256. When thedisplay control for a gesture is started, in step S201, the gesturedetector 253 starts receiving a gesture performed by the occupant. Atthis time, a center information display 13 c displays a messageproposing to start a hands-free call. After the receiving of the gestureis started, the control proceeds to step S211.

In step S211, the display controller 70 determines whether there is aninput of a gesture by the occupant. The gesture can be set as, forexample, shaking the palm of a hand to the gesture detector 253. Morespecifically, shaking the palm in the up-down direction can be set asthe gesture to start a hands-free call in response to the proposal ofthe hands-free call. On the other hand, shaking the palm in theright-left direction can be set as the gesture not to start a hands-freecall.

When there is the gesture to start a hands-free call, the hands-freecall is started and the process proceeds to step 212. When there is thegesture not to start a hands-free call, the call is rejected and theprocess proceeds to step 212. On the other hand, when there is nogesture for a certain period of time, the process proceeds to step 261.

In step S212, the display controller 70 determines whether the vehicleis stopped. When the vehicle is determined not to be stopped, i.e.,determined to be traveling in step S212, the control proceeds to stepS221. On the other hand, when the vehicle is determined to be stopped instep S212, the process proceeds to step S261.

In step S221, an area acquisition unit 72 acquires the vision area VA ofthe driver. The area acquisition unit 72 also acquires a display area.After the vision area VA and the latest display area are acquired, thecontrol proceeds to step S222. The control in step S221 provides anexample of an area acquisition step.

In step S222, the position determination unit 73 determines whether thedisplay area is included in the vision area VA. When even a part of thedisplay area is determined to be included in the vision area VA in stepS222, the control proceeds to step S224. On the other hand, when thedisplay area is determined not to be included in the vision area VA instep S222, it is determined that the driver cannot immediately visuallyrecognize information displayed by the display 10, and the controlproceeds to step S251. The control of step S222 provides an example of aposition determination step.

In step S224, the position determination unit 73 determines whether oneof the display screens of the electronic mirrors 15 is included in thevision area VA. FIG. 9 shows the vision area VA when the driver isgazing at a right electronic mirror 15. The center position of thevision area VA is located on the display screen of the electronic mirror15. The display area included in the vision area VA is only the displayscreen of the electronic mirror 15. The control of step S224 provides anexample of a position determination step.

When the display screen of the electronic mirror 15 is included in thevision area VA, it can be assumed that the driver is particularlyconcentrated on driving. Therefore, the process returns to step S221without displaying on the display screen of the electronic mirror 15. Asa result, the display of the gesture operation result is stopped untilthe driver visually recognizes the display area other than theelectronic mirror 15 or the display area is no longer included in thevision area VA. On the other hand, when the display screen of theelectronic mirror 15 is determined not to be included in the vision areaVA in step S224, the process proceeds to step S231.

In step S231, the position determination unit 73 determines whether thedisplay screen of the center information display 13 c is included in thevision area VA. When the center information display 13 c is determinedto be included in the vision area VA in step S231, the control proceedsto step S261. On the other hand, when the display screen of the centerinformation display 13 c is determined not to be included in the visionarea VA in step S231, the control proceeds to step S241. The control ofstep S231 provides an example of a position determination step.

In step S241, the display execution unit 74 displays the gestureoperation result on the display screen included in the vision area VA.For example, when the vision area VA is located on the display portion11 d of the head-up display 11, the gesture operation result isdisplayed as an icon on the display portion 11 d. Here, the icon is adisplay indicating that the hands-free call is in progress or a displayindicating that the call is rejected.

For example, when the vision area VA is located on the display screen ofthe meter display 13 m, the gesture operation result is displayed as anicon on the display screen of the meter display 13 m. For example, whenthe vision area VA is located on a right portion of the infotainmentdisplay 13 i, the gesture operation result is displayed as the icon onthe right portion of the infotainment display 13 i. After the icon ofthe gesture operation result is displayed on the display screen in thevision area VA in step S241, the control proceeds to step S261. Thecontrol of the step S241 provides an example of a display executionstep.

In step S251, the display execution unit 74 outputs the gestureoperation result by sound. For example, notification of the start of thehands-free call is output from the speaker 21 as a notification sound.Alternatively, notification of the rejection of the call is output fromthe speaker 21 as a notification sound. After outputting the gestureoperation result by sound at S251, the control proceeds to step S261.

In step S261, the display execution unit 74 displays the gestureoperation result on the center information display 13 c. The centerinformation display 13 c displays detailed information instead of anicon. When the hands-free call is started, the detailed displayincludes, for example, information on the other party of the hands-freecall and information on the elapsed time from the start of thehands-free call. When the hands-free call is not started, the detaileddisplay includes, for example, information on the other party of thecall and information on the time when the call is received. This casewhere the hands-free call is not started includes both a case where thehands-free call is rejected by the gesture and a case where the gestureis not performed. After the detailed information is displayed on thecenter information display 13 c in step S261, the control proceeds tostep S291. The control of the step S261 provides an example of a displayexecution step.

In step S291, the display controller 70 determines whether the receivingof the gesture has ended. When the receiving of the gesture has ended,it is determined that there is no possibility that a new gesture isinput by the occupant, and the display control for the gesture ends. Onthe other hand, when the receiving of the gesture is maintained, it isdetermined that there is a possibility that a new gesture is input bythe occupant, and the control returns to step S211 to repeat the seriesof display controls for the gesture.

The following will describe technical effects of the present embodiment.According to the embodiment described above, when the electronic mirror15 corresponds to the display area included in the vision area VA, thedisplay execution unit 74 suspends the display of the gesture operationresult until the electronic mirror 15 is no longer included in thevision area VA. Therefore, the gesture operation result can be preventedfrom being displayed in a situation where the driver is presumed toconcentrate on driving.

Although the air conditioner or the mobile terminal 256 via thecommunication device 255 has been described as an example of theoperation target device of the operation by gesture, the operationtarget device is not limited to the example described above. Forexample, a car navigation device or an in-vehicle lighting device may beused as an operation target device, and the control of the volume ofroute guidance or illuminance may be configured to be operable by agesture.

The disclosure in this specification, the drawings, and the like is notlimited to the exemplified embodiments. The disclosure encompasses theillustrated embodiments and variations based on the embodiments by thoseskilled in the art. For example, the present disclosure is not limitedto the combinations of components and/or elements shown in theembodiments. The disclosure may be implemented in various combinations.The disclosure may have additional members which may be added to theembodiments. The disclosure encompasses omission of components and/orelements of the embodiments. The disclosure encompasses the replacementor combination of components and/or elements between one embodiment andanother. The disclosed technical scope is not limited to thedescriptions of the embodiments.

The controller and the method described in the present disclosure may beimplemented by a special purpose computer configuring a processorprogrammed to perform one or more functions embodied by a computerprogram. Alternatively, the apparatuses and methods described in thisapplication may be fully implemented by special purpose hardware logiccircuits. Further alternatively, the apparatuses and methods describedin this application may be implemented by a special purpose computercreated by a combination of a processor executing computer programscoupled with hardware logic circuits. The computer program may be storedin a computer-readable non-transition tangible storage medium as aninstruction executed by a computer.

What is claimed is:
 1. A display system configured to change display inaccordance with a gesture of an operator in a vehicle, the displaysystem comprising: a gesture detector configured to detect the gesture;a vision area detector configured to detect a vision area of theoperator; a display configured to display a gesture operation resultthat is an operation result of the gesture; and a display controllerconfigured to execute a control related to display processing of thedisplay, wherein the display includes a center information displayhaving a touch panel capable of receiving a touch operation, and thedisplay controller includes a processor configured to: acquire thevision area detected by the vision area detector and a display area ofthe display, determine a positional relationship between the vision areaand the display area, and display the gesture operation result to theoperator, determine a display mode of the gesture operation result basedon the positional relationship between the vision area and the displayarea, and display the gesture operation result on the center informationdisplay regardless of the positional relationship between the visionarea and the display area when the vehicle is stopped.
 2. The displaysystem according to claim 1, wherein the display includes an electronicmirror for monitoring surroundings of the vehicle, and the processor isfurther configured to stop the display of the gesture operation resultuntil the electronic mirror is no longer included in the vision areawhen the electronic mirror has the display area included in the visionarea.
 3. A display system configured to change display in accordancewith a gesture of an operator in a vehicle, the display systemcomprising: a gesture detector configured to detect the gesture; avision area detector configured to detect a vision area of the operator;a display configured to display a gesture operation result that is anoperation result of the gesture; and a display controller configured toexecute a control related to display processing of the display, whereinthe display includes an electronic mirror for monitoring surroundings ofthe vehicle, and the display controller includes a processor configuredto acquire the vision area detected by the vision area detector and adisplay area of the display, determine a positional relationship betweenthe vision area and the display area, and display the gesture operationresult to the operator, determine a display mode of the gestureoperation result based on the positional relationship between the visionarea and the display area, and stop the display of the gesture operationresult until the electronic mirror is no longer included in the visionarea when the electronic mirror has the display area included in thevision area.
 4. The display system according to claim 1, wherein theprocessor is further configured to display the gesture operation resultin the display area within the vision area when at least a part of thedisplay area is included in the vision area, and the processor isfurther configured not to display the gesture operation result in thedisplay area when the display area is not included in the vision area.5. The display system according to claim 4, wherein the display iscapable of displaying the gesture operation result at an arbitraryposition in the display area, and the processor is further configured todisplay the gesture operation result at a position including a centerposition of the vision area.
 6. The display system according to claim 4,wherein the display includes display sections capable of displaying thegesture operation result, and the processor is further configured todisplay the gesture operation result in a display section located at aposition closest to a center position of the vision area among thedisplay sections.
 7. The display system according to claim 1, whereinthe display includes a small display, and a large display larger indisplay area than the small display, and the processor is furtherconfigured to display different contents between a case of displayingthe gesture operation result on the small display and a case ofdisplaying the gesture operation result on the large display.
 8. Thedisplay system according to claim 7, wherein the small display iscapable of displaying a virtual image superimposed on a front window ofthe vehicle, the large display includes display screens provided on aninstrument panel and arranged next to each other, the processor isfurther configured to display an icon when the gesture operation resultis displayed using the small display, and the processor is furtherconfigured to display detailed information when the gesture operationresult is displayed using the large display.
 9. The display systemaccording to claim 8, wherein the large display includes a centerinformation display having a touch panel as one of the display screens,which functions as the gesture detector, and the processor is furtherconfigured to display the gesture operation result on the centerinformation display in priority to others of the display screens whenthe center information display is included in the vision area.
 10. Thedisplay system according to claim 1, further comprising a speakerconfigured to output sound into the vehicle, wherein the processor isfurther configured to output the gesture operation result from thespeaker as sound without displaying the gesture operation result usingthe display when the display area is not included in the vision area.